Motion vector estimation device, motion vector estimation method, and program for estimating motion vector

ABSTRACT

Provided is a motion vector estimation device capable of estimating the motion vector with less computation. A motion vector estimation device for estimating, by means of repetitive calculations, the motion vector of each of a plurality of pixel groups which is contained in an input image and which each contains one or more pixels, the motion vector estimation device being provided with a means for making repetitive calculations with regard to the pixel groups that do not have a high frequency component from among the plurality of pixel groups contained in the input image after making repetitive calculations with regard to the pixel groups that have a high frequency component from among the plurality of pixel groups contained in the input image.

This application is a National Stage Entry of PCT/JP2012/077554 filed onOct. 25, 2012, which claims priority from Japanese Application2011-249679 filed on Nov. 15, 2011, the contents of all of which areincorporated herein by reference, in their entirety.

TECHNICAL FIELD

The present invention relates to a motion vector estimation device forestimating a motion vector from a moving image, a motion vectorestimation method, and a program for estimating motion vector.

BACKGROUND ART

Processing of estimating a motion of each pixel between consecutiveframes in a moving image is used in an MPEG (Moving Picture ExpertsGroup) coding method, and device, a three-dimensional noise removalmethod, and device that remove noise by position-aligning images of aplurality of frames and combining the images, and a super resolutiontechnique for generating a high resolution image from images of aplurality of frames.

Two luminance images f and f′ having a predetermined time intervaltherebetween, in a moving image including a minute motion, and a resultobtained by estimating a motion vector from f to f′ in each pixel areillustrated in FIG. 1. Hereafter, a horizontal direction component of amotion vector at a coordinate (x, y) is denoted by u(x, y), and avertical direction component thereof is denoted by v(x, y).

An example of a conventional motion vector estimation method isdescribed in NPL 1. In the technique, an energy function E representedby the following Math. is considered.

Math. 1

$E = {\sum\limits_{x,{y \in \; f}}\left\{ {\left( {{{f_{x}\left( {x,y} \right)} \cdot {u\left( {x,y} \right)}} + {{f_{y}\left( {x,y} \right)} \cdot {v\left( {x,y} \right)}} + {f_{t}\left( {x,y} \right)}} \right)^{2} + {\alpha\left( {{{\bigtriangledown\;{u\left( {x,y} \right)}}}^{2} + {{\bigtriangledown\;{v\left( {x,y} \right)}}}^{2}} \right)}} \right\}}$

Here, the first term is called data term, and the second term is calledregularizing term. Furthermore, f_(x)(x, y), f_(y)(x, y), and f_(t)(x,y) are partial differentials of a pixel value f(x, y) at a coordinate(x, y) in directions of an x axis, a y axis, and a time axis,respectively, and are represented by the following Math.f _(y)(x,y)=f(x,y+1)−f(x,y)f _(x)(x,y)=f(x+1,y)−f(x,y)f _(t)(x,y)=f′(x,y)−f(x,y)  Math. 2

Furthermore,∇u(x,y),∇v(x,y)  Math. 3are gradient vectors at the coordinate (x, y) and represented by thefollowing Math.

Math. 4

${\bigtriangledown\;{u\left( {x,y} \right)}} = \begin{pmatrix}{{u\left( {{x + 1},y} \right)} - {u\left( {x,y} \right)}} \\{{u\left( {x,{y + 1}} \right)} - {u\left( {x,y} \right)}}\end{pmatrix}$${\bigtriangledown\;{v\left( {x,y} \right)}} = \begin{pmatrix}{{v\left( {{x + 1},y} \right)} - {v\left( {x,y} \right)}} \\{{v\left( {x,{y + 1}} \right)} - {v\left( {x,y} \right)}}\end{pmatrix}$

The first term in { } in the right side of Math. 4 is called data termand represents a constraint that the luminance value on the images f andf′ does not change between before and after a movement with motionvectors u(x, y) and v(x, y). In the same way, the second term is calledsmoothing term, and represent a constraint that the motion vectors u(x,y) and v(x, y) change with spatial smoothness. Intensities of bothconstraints are adjusted by using a smoothing term weight a.

Optimum motion vectors u(x, y) and v(x, y) minimize the above-describedenergy function. At this time, the following constraint Math. concerningu(x, y) and v(x, y) is obtained by making a partial differential of theabove-described energy function with respect to u(x, y) and v(x, y)equal to zero.f _(s)(x,y)² ·u(x,y)+f _(x)(x,y)·f _(y)(x,y)·v(x,y)+f _(x)(x,y)·f_(t)(x,y)−α·Δu(x,y)=0f _(s)(x,y)·f _(y)(x,y)·u(x,y)+f _(y)(x,y)² ·v(x,y)+f _(y)(x,y)·f_(t)(x,y)−α·Δv(x,y)=0  Math. 5

Here, Δ is the Laplacian, and Δ_(u)(x, y) and Δ_(v)(x, y) arerepresented by the following Math.Δu(x,y)=u(x+1,y)+u(x−1,y)+u(x,y+1)+u(x,y−1)−4·u(x,y)Δv(x,y)=v(x+1,y)+v(x−1,y)+v(x,y+1)+v(x,y−1)−4·v(x,y)  Math. 6

The above-described constraint Math. includes equations concerning themotion vectors u(x, y) and v(x, y) at each coordinate, and u(x, y) andv(x, y) are found by solving the simultaneous equations.

By the way, since repetitive calculations need a long time, a techniquefor detecting a motion vector without using repetitive calculations isdescribed in PTL 1.

A technique concerning a motion detection circuit is described in PTL 2.However, the technique only detects whether there is a motion and cannotdetect a motion vector.

A technique of changing over a gradient method or a block matchingmethod depending upon the number of detected gradient parts anddetecting a motion vector is described in PTL 3. However, this techniquefinds only one motion vector for an image as a whole.

A technique of detecting a motion vector from a contracted image andthen detecting a motion vector of an image having an original resolutionis described in PTL4. However, this technique relates to improvement ofa search range of a motion vector.

A technique of determining a hierarchy in which motion detection isstarted by using data obtained by conducting discrete waveletdecomposition on an image, in a method for hierarchically detecting amotion vector is described in PTL 5.

CITATION LIST Patent Literature

-   {PTL 1} JP-A-6-150007-   {PTL 2} JP-A-2000-115585-   {PTL 3} JP-A-2009-88884-   {PTL 4} JP-A-2010-74496-   {PTL 5} JP-A-2011-82700

Non Patent Literature

-   {NPL 1} Bruhn et al., “Lucas/Kanade meets Horn/Schunck: combining    local and global optic flow methods,” International Journal of    Computer Vision, Volume 61 Issue 3, 2005.

SUMMARY OF INVENTION Technical Problem

A problem of the technique in the above-described NPL 1 is that thequantity of calculations is large. The reason is as follows. It ispractically impossible to solve the above-described simultaneousequations analytically because giant matrix computations of (the numberof pixels×2)×(the number of pixels×2) dimension are needed. In general,it is necessary to give initial values to u(x, y) and v(x, y) andoptimize them by repetitive calculations.

An object of the present invention is to provide a motion vectorestimation device, a motion vector estimation method, and a motionvector estimation program capable of estimating a motion vector with aless calculation quantity.

Solution to Problem

According to a first aspect, the present invention provides a motionvector estimation device that estimates a motion vector for each of aplurality of pixel groups included in an input image, each pixel groupincluding at least one pixel, by repetitive calculations, the motionvector estimation device including a means that conducts the repetitivecalculations intended for pixel groups having a high frequency componentamong the plurality of pixel groups included in the input image, andthen conducts the repetitive calculations intended for pixel groupshaving no high frequency components among the plurality of pixel groupsincluded in the input image.

According to a second aspect, the present invention provides a motionvector estimation method used to estimate a motion vector for each of aplurality of pixel groups included in an input image, each pixel groupincluding at least one pixel, by repetitive calculations, the motionvector estimation method including a step of conducting the repetitivecalculations intended for pixel groups having a high frequency componentamong the plurality of pixel groups included in the input image, andthen conducting the repetitive calculations intended for pixel groupshaving no high frequency components among the plurality of pixel groupsincluded in the input image.

According to a third aspect, the present invention provides a motionvector estimation program for causing a computer to function as a motionvector estimation device that estimates a motion vector for each of aplurality of pixel groups included in an input image, each pixel groupincluding at least one pixel, by repetitive calculations, the motionvector estimation program causing the computer to function as a meansthat conducts the repetitive calculations intended for pixel groupshaving a high frequency component among the plurality of pixel groupsincluded in the input image, and then conducts the repetitivecalculations intended for pixel groups having no high frequencycomponents among the plurality of pixel groups included in the inputimage.

Advantages Effects of the Invention

According to the present invention, it is possible to estimate a motionvector with a less calculation quantity.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A figure illustrates a diagram for explaining a motion vectortherebetween two luminance images with a predetermined time intervalbetween in a moving image with respect to each pixel.

FIG. 2 A figure illustrates a diagram for explaining the number of timesof repetitive calculations every pixel kind conducted in an embodimentof the present invention.

FIG. 3 A figure is a block diagram illustrating a configuration of amotion vector estimation device according to an embodiment 1 of thepresent invention.

FIG. 4 A figure is a flow chart illustrating a motion vector estimationmethod according to the embodiment 1 of the present invention.

FIG. 5 A figure illustrates an example of a high frequency mask imageutilized in an embodiment of the present invention.

FIG. 6 A figure is a block diagram illustrating a configuration of amotion vector estimation device according to an embodiment 2 of thepresent invention.

FIG. 7 A figure is a flow chart illustrating a motion vector estimationmethod according to the embodiment 2 of the present invention.

FIG. 8 A figure illustrates a concept diagram for explaining a motionvector estimation method according to the embodiment 2 of the presentinvention.

FIG. 9 A figure is a block diagram illustrating a configuration of aninterlace-progressive conversion device according to an embodiment 3 ofthe present invention.

FIG. 10 A figure illustrates a first concept diagram for explaining aninterlace-progressive conversion method according to the embodiment 3 ofthe present invention.

FIG. 11 A figure illustrates a second concept diagram for explaining theinterlace-progressive conversion method according to the embodiment 3 ofthe present invention.

DESCRIPTION OF EMBODIMENTS

Hereafter, embodiments of the present invention will be described indetail with reference to the drawings.

In the present embodiment, attention is paid to a point that the numberof repetitions required until values of motion vectors u(x, y) and v(x,y) converge differs depending upon a pixel in a process of optimizationusing repetitive calculations. Specifically, as for a pixel having ahigh frequency on an image (such as a pixel located near an edge), thevalues converge in an initial stage of the repetitive calculations. Onthe other hand, as for a pixel having a low frequency (such as a pixelin a flat area that is less in luminance change), convergence beginslater as compared with high frequency pixels. In the present embodiment,this characteristic is used. As illustrated in FIG. 2, the whole of therepetitive calculations is divided into a plurality of stages. In eachstage, repetitive calculations are performed on only pixels for whichconvergence in the stage is expected. As a result, the object of thepresent invention can be achieved.

In other words, no matter whether the repetitive calculations areconducted for all pixels or the repetitive calculations are conductedfor only high frequency pixels, the number of times of repetitionrequired until calculations for high frequency pixels converge in theearly stage of repetitive calculations changes little. First, therefore,repetitive calculations are conducted for only high frequency pixels. Bydoing so, calculations for low frequency pixels are excluded from therepetitive calculations. As a result, it is possible to reduce thenumber of pixels that become objects of calculation per repetition.

A state at the time when repetitive calculations for only high frequencypixels have converged is nearly the same as a state at the time whenrepetitive calculations for all pixels have converged for only the highfrequency pixels. If repetitive calculations for only high frequencypixels have converged, therefore, then repetitive calculations areconducted for only low frequency pixels. A convergence process inrepetitive calculations for only low frequency pixels conducted afterrepetitive calculations are conducted for only high frequency pixels isnearly the same as a convergence process in repetitive calculationsconducted substantially for only low frequency pixels conductedsubsequently to first convergence substantially for only high frequencypixels in a case where repetitive calculations for all pixels areconducted. If repetitive calculations for only high frequency pixelshave converged, therefore, no problem is posed at all even if repetitivecalculations are then conducted for only low frequency pixels.Calculations for high frequency pixels are excluded from repetitivecalculations for only low frequency pixels. As a result, it is possibleto reduce the number of pixels that become objects of calculation perrepetition.

Even if the total number of repetitions in the present embodiment is thesame as the number of repetitions in the conventional calculation,therefore, the whole calculation quantity can be reduced because it ispossible to reduce the number of pixels that become objects ofcalculation per repetition. Furthermore, a state of calculationconvergence in a case where repetitive calculations are first conductedfor only high frequency pixels and then repetitive calculations areconducted for only low frequency pixels hardly changes from a state ofcalculation convergence in a case where repetitive calculations areconducted for all pixels from the beginning to the end. Therefore, thetotal number of repetitions in the case where repetitive calculationsare first conducted for only high frequency pixels and then repetitivecalculations are conducted for only low frequency pixels hardlyincreases as compared with the case where repetitive calculations areconducted for all pixels from the beginning to the end. In the presentembodiment, therefore, the whole calculation quantity can be reduced.

By the way, repetitive calculations are conducted for high frequencypixels, and repetitive calculations are conducted for low frequencypixels, and finally repetitive calculations are conducted for allpixels. The reason is that processing uniting adjacent pixels isconducted by finally conducting repetitive calculations for all pixelsand thereby a motion vector estimated at a boundary between a highfrequency pixel and a low frequency pixel is provided with continuity.

Embodiment 1

An image processing device according to an embodiment 1 is illustratedin FIG. 3. Furthermore, the whole of an image processing methodconducted by the image processing device is illustrated in FIG. 4.

In the present embodiment, the whole of repetitive calculations isdivided into a plurality of stages. In each stage, repetitivecalculations are applied to only pixels for which convergence in eachstage is expected. When the number of times of repetition determined foreach stage is reached, or when a difference between a result ofrepetitive calculations in each pixel obtained last time and a result ofrepetitive calculations in the pixel obtained this time has become apredetermined threshold or less, repetitive calculations arediscontinued and processing proceeds to the next stage. As a result, amotion vector estimation result having a precision approximatelyequivalent to that of the conventional technique is obtained with acalculation quantity less than that of the conventional technique. Here,a result of repetitive calculations is, for example, a value of theenergy function. Furthermore, as a method of repetitive calculations,for example, the gradient method, the conjugate gradient method, theGauss-Newton method, or the Levenberg-Marquardt method is used.

With reference to FIG. 3, a motion vector estimation device 101according to the embodiment 1 includes a high frequency/low frequencydecision unit 103, a number of times of repetition determination unit105. a partial differential coefficient calculation unit 107, and amotion vector estimation unit 109.

The high frequency/low frequency decision unit 103 makes a decisionwhether each of pixels in a current frame f includes a high frequency.If the pixel includes a high frequency, the high frequency/low frequencydecision unit 103 judges the pixel to be a high frequency pixel. Unlessthe pixel includes a high frequency, the high frequency/low frequencydecision unit 103 judges the pixel to be a low frequency pixel. Here,“the pixel includes a high frequency” means that an output level of atleast a predetermined value is obtained when a spatial filter thatpasses high frequencies and obstructs low frequencies is disposed aroundthe pixel. The high frequency/low frequency decision unit 103 outputs ahigh frequency mask image as illustrated in FIG. 5 on the basis ofdecision results of respective pixels. In FIG. 5, a white colored partis formed of high frequency pixels and a black colored part is formed oflow frequency parts.

The number of times of repetition determination unit 105 calculates i₁and i₂ for determining the number of times of repetition i₁ inrepetitive calculations for high frequency pixels, the number of timesof repetition i₂−i₁ in repetitive calculations for low frequency pixels,and the number of times of repetition i_(MAX)−i₂ in repetitivecalculations for all pixels, on the basis of a ratio of high frequencypixels to all pixels included in a high frequency mask image, aparameter having a predetermined value, and so on. Specifically, thenumber of times of repetition determination unit 105 finds i₁ and i₂according toi=ratio_(high)·(i _(MAX) −n)i ₂ =i _(MAX) −nwhere ratio_(high) is a ratio of high frequency pixels to the wholeimage, n is a predetermined parameter, and i_(MAX) is the total numberof times of repetition. However, ratio_(high) may also be apredetermined parameter.

The partial differential coefficient calculation unit 107 calculates apartial differential coefficient f_(x)(x, y) of a motion vector for eachpixel value f(x, y) in the x direction, a partial differentialcoefficient f_(y)(x, y) of the motion vector for each pixel in the ydirection, and a partial differential coefficient f_(t)(x, y) of themotion vector for each pixel in the time direction on the basis of apixel included in a current frame image f and a pixel included in thenext frame image f′.

The motion vector estimation unit 109 receives the high frequency maskimage, the number of times of repetition i₁, i₂ and i_(MAX), and thepartial differential coefficients f_(x)(x, y), f_(y)(x, y) and f_(t)(x,y) as inputs, and estimates motion vectors u(x, y) and v(x, y) for eachhigh frequency pixel in the current frame image f and motion vectorsu(x, y) and v(x, y) for each low frequency pixel in the current frameimage f on the basis of the inputs. For a pixel judged to be a highfrequency pixel on the basis of the high frequency mask image, acalculation according to a solution of repetitive calculations forsolving the above-described constraint Math is repeated the number oftimes specified by the number of times of repetition i₁. For a pixeljudged to be a low frequency pixel on the basis of the high frequencymask image, a calculation according to a solution of repetitivecalculations for solving the above-described constraint Math is repeatedthe number of times specified by the number of times of repetitioni₂−i₁. In addition, for all pixels, a calculation according to asolution of repetitive calculations for solving the above-describedconstraint Math is repeated the number of times specified by the numberof times of repetition i_(MAX)−i₂.

An operation of the motion vector estimation device illustrated in FIG.3 will now be described with reference to FIG. 4. With reference to FIG.4, first, the high frequency/low frequency decision unit 103 determineswhether each of pixels in the current frame image f is a high frequencypixel or a low frequency pixel (step S201).

Then, the number of times of repetition determination unit 105determines (calculates) the number of times i₁, i₂ and i_(MAX) relatingto the number of times of repetition (step S203).

Then, the partial differential coefficient calculation unit 107calculates partial differential coefficients f_(x)(x, y), f_(y)(x, y)and f_(t)(x, y) for each pixel value f(x, y) in the current frame image(step S205).

Then, for high frequency pixels included in the current frame image, therepetitive calculation for solving the above-described constraint Mathto find a motion vector is repeated i₁ times. For low frequency pixelsincluded in the current frame image, the repetitive calculation forsolving the above-described constraint Math to find a motion vector isrepeated i₂−i₁ times. And for all frequency pixels included in thecurrent frame image, the repetitive calculation for solving theabove-described constraint Math to find a motion vector is repeatedi_(MAX)−i₂ times. (steps S207 and S208). As for a pixel for which acalculation result has converged in the middle of each repetition,however, it is not necessary to conduct repetitive calculationsthereafter.

Embodiment 2

An embodiment 2 will now be described in detail with reference to thedrawings.

The embodiment 2 provides the embodiment 1 with multiple resolutions.

An image processing device according to the embodiment 2 is illustratedin FIG. 6. Furthermore, the whole of an image processing methodconducted by the image processing device is illustrated in FIG. 7.

In the embodiment 2, a motion vector is found in a low resolution imageobtained by downscaling an original image to ¼ in the longitudinaldirection and ¼ in the lateral direction. And at the next resolutionlevel of ½ in the longitudinal direction and ½ in the lateral direction,a motion vector obtained by upscaling the motion vector found at theresolution level of ¼ in the longitudinal direction and ¼ in the lateraldirection, according to the resolution level of ½ in the longitudinaldirection and ½ in the lateral direction is set to be an initial valueof a motion vector. Then, a motion vector at a resolution level of ½ inthe longitudinal direction and ½ in the lateral direction is found. Suchprocessing is repeated up to a final resolution level, i.e., up to thesame resolution level as that of the input image. It also becomespossible to estimate a large motion vector that cannot be estimatedcorrectly in the embodiment 1, by taking such a configuration.

With reference to FIG. 6, a motion vector estimation device 101Baccording to the embodiment 2 includes a high frequency/low frequencydecision unit 103B, a number of times of repetition determination unit105B, a partial differential coefficient calculation unit 107B, a motionvector estimation unit 109B, a resolution pyramid creation unit 111, anda motion vector upscaling unit 113.

The resolution pyramid creation unit 111 creates a primary lowresolution image having a resolution of ½ in the longitudinal directionand ½ in the lateral direction and a secondary low resolution imagehaving a resolution of ¼ in the longitudinal direction and ¼ in thelateral direction for each of the current frame image f and the nextframe image f′. When creating an image having a resolution of ½ in thelongitudinal direction and ½ in the lateral direction as compared withan image having a resolution, from the image having the resolution, highfrequency components are removed by applying a Gaussian filter andpixels are sampled every other pixel in the longitudinal direction andthe lateral direction. However, this is a method in a case where aresolution that is lower than a certain resolution by one stage is ½ ascompared with the certain resolution. The resolution that is lower thanthe certain resolution by one stage may be other than ½. In that case,for example, filtering and resampling depending upon the resolution areconducted.

The high frequency/low frequency decision unit 103B, the number of timesof repetition determination unit 105B, the partial differentialcoefficient calculation unit 107B and the motion vector estimation unit109B are similar to the high frequency/low frequency decision unit 103,the number of times of repetition determination unit 105, the partialdifferential coefficient calculation unit 107 and the motion vectorestimation unit 109, respectively. For an image having the sameresolution as the input image (original resolution image), the primarylow resolution image, and the secondary low resolution image, however,the high frequency/low frequency decision unit 103B, the number of timesof repetition determination unit 105B, the partial differentialcalculation unit 107B and the motion vector estimation unit 109B operatein order of the secondary resolution image, the primary low resolutionimage, and the original resolution image.

The motion upscaling unit 113 upscales a motion vector found for a pixelincluded in an image of the current resolution to a resolution of twicein the longitudinal direction and twice in the lateral direction byusing a predetermined method (such as, for example, bilinearinterpolation, nearest interpolation, bicubic interpolation, and so on),and feeds back the upscaled motion vector to the motion vectorestimation unit 109B as an initial value to be used when finding amotion vector for a pixel included in a pixel of the next resolution(twice in the longitudinal direction and twice in the lateraldirection).

An operation of the motion vector estimation device 101B illustrated inFIG. 6 will now be described with reference to FIG. 7.

Since steps S201, S203, S205, S207 and S209 are similar to the stepsS201, S203, S205, S207 and S209 in the embodiment, respectively,duplicated description will be omitted. However, i_(MAX) is made todiffer every resolution r and represented by i^(r) _(MAX). Correspondingto this, i₁ and i₂ are replaced by i^(r) ₁ and i^(r) ₂.

First, the resolution pyramid creation unit 111 creates the originalresolution image, the primary low resolution image, and the secondarylow resolution image for each of the current frame image f and the nextframe image f′(step S221).

Then, steps S201, S203, S205, S207 and S209 are executed.

Then, the motion vector upscaling unit 113 upscales a motion vector(step S223).

Then, it is determined whether the current resolution is the finalresolution (i.e., the original resolution) (step S225). Unless thecurrent resolution is the final resolution (NO at the step S225), theresolution is advanced by one stage (the resolution is increased totwice in the longitudinal direction and twice in the lateral direction),and the processing returns to the step S201. If the current resolutionis the final resolution (YES at the step S225), the motion vector foundat the present time is taken as the final estimated motion vector andthe processing is terminated.

Embodiment 3

An embodiment 3 will now be described in detail with reference to thedrawings.

The embodiment 3 is an application of the motion vector estimationdevice 101 according to the embodiment 1 or the motion vector estimationdevice 101B according to the embodiment 2 to an interlace-progressiveconversion device.

With reference to FIG. 9, an interlace-progressive conversion device 121according to the embodiment 3 includes the motion vector estimationdevice 101 or 101B, a motion compensation unit 123, and a fieldcombination unit 125.

As illustrated in FIG. 10, the motion vector estimation device 101 or101B regards an odd-numbered field image or an even-numbered field imageincluded in interlace images as a field image that becomes an object ofmotion vector estimation, finds the partial differential coefficientsf_(x)(x, y), f_(y)(x, y) and f_(t)(x, y) in the above-describedconstraint Math. for calculating a motion vector on the basis of twofield images having the object field image therebetween, solves theabove-described constraint Math. using the partial differentialcoefficients by conducting repetitive calculations, and therebyestimates a motion vector for a pixel included in the object fieldimage. By the way, the motion vector is a motion vector between thefield image that becomes the object of the motion vector estimation anda field image adjacent to the object field image.

The motion compensation unit 123 conducts motion compensation on a pixelincluded in the field image that has become the object of the motionvector estimation, by using the motion vector estimated by the motionvector estimation device 101 or 101B.

As illustrated in FIGS. 10 and 11, the field combination unit 125combines the field image subjected to the motion compensation conductedby the motion compensation unit 123 with the other field image includedin the input interlace image, thereby obtains a progressive image, andoutputs the progressive image.

The embodiments 1 to 3 bring about an effect that it is possible toobtain an estimated motion vector having a precision nearly equal tothat of the conventional technique with a calculation quantity less thanthat of the conventional technique.

The reason is that the whole of repetitive calculations is divided intoa plurality of stages and in each stage only pixels for whichconvergence is expected in the stage are set to be an object ofrepetitive calculations.

Example 1

The above-described embodiments will now be described by using concreteexamples. The present example relates to the embodiment 2.

First, the resolution pyramid creation unit 111 illustrated in FIG. 6generates a resolution pyramid for each of the current frame image f andthe next frame image f′. Specifically, the resolution pyramid creationunit 111 applies a Gaussian filter to each (hereafter referred to as“original frame image”) of the current frame image f and the next frameimage f′, thereby removes high frequency components, then conductssampling on pixels every other pixel, and thereby generates a primarylow resolution image having an image resolution that becomes ½ in thelongitudinal direction and ½ in the lateral direction as compared withthe original frame image.

In addition, the resolution pyramid creation unit 111 applies theabove-described Gaussian filter and the sampling every other pixel tothe primary resolution frame image, and thereby generates a secondarylow resolution image having an image resolution that becomes ¼ in thelongitudinal direction and ¼ in the lateral direction as compared withthe original frame image.

Hereafter, resolutions that are ¼ times, ½ times and equal as comparedwith the resolution of the original image are referred to as resolutionlevels 1, 2 and 3, respectively.

Subsequently, processing described hereafter is repeated in order of theresolution level 1, 2 and 3. First, the high frequency/low frequencydecision unit 103B determines whether each of pixels in the currentframe image f having a current resolution level r (r=1, 2, 3) has a highfrequency component, and generates a high frequency mask imagerepresenting pixels having a high frequency component. Specifically, thehigh frequency/low frequency decision unit 103B applies Sobel filters inthe horizontal and vertical directions represented by a coefficientmatrix described below to the current frame image f, calculatesluminance gradient components in the horizontal and vertical directionsat each coordinate, and judges a pixel for which a luminance gradientintensity found on the basis of luminance gradient components in thehorizontal and vertical directions exceeds a threshold to be a highfrequency pixel. And a pixel that is not a high frequency pixel isjudged to be a low frequency pixel. Here, for example, a square sum, anabsolute value sum, or a maximum value of a luminance gradient componentin the horizontal direction and a luminance gradient component in thevertical direction is taken as a luminance gradient.

Math. 7

$\begin{pmatrix}{- 1} & 0 & 1 \\{- 2} & 0 & 2 \\{- 1} & 0 & 1\end{pmatrix}\mspace{14mu}\begin{pmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\1 & 2 & 1\end{pmatrix}$

By doing so, a pixel having a low frequency component and having no highfrequency components is classified into low frequency components,whereas a pixel having no low frequency components and having a highfrequency component, and a pixel having a high frequency component andhaving a high frequency component are classified into high frequencycomponents.

Then, the number of times of repetition determination unit 105Bcalculates the number of times of repetition i^(r) ₁, i^(r) ₂ and i^(r)_(MAX) for terminating the stages 1, 2 and 3, respectively, at thecurrent resolution level r. First, the number of times of repetitiondetermination unit 105B calculates i^(r) _(MAX) according to thefollowing expression.i ^(r) _(MAX)=β^(r-1) ·i _(MAX)where i^(r) _(MAX) is the predetermined maximum number of times ofrepetition, and β is a parameter (0<β<1) that attenuates the maximumnumber of times of repetition as the resolution level advances. Andβ^(r-1) represents the (r−1)th power of β. Subsequently, the number oftimes of repetition determination unit 105B calculates the number oftimes of repetition i^(r) ₁ for terminating the stage 1, i.e., the stagefor optimizing an estimated motion concerning a high frequency pixel, atthe current resolution level r, and the number of times of repetitioni^(r) ₂ for terminating the stage 2, i.e., the stage for optimizing anestimated motion concerning a low frequency pixel, at the currentresolution level r, according to the following expressions.i ^(r) ₁=ratio_(high)·(i ^(r) _(MAX) −n)i ^(r) ₂ =i ^(r) _(MAX) −nwhere ratio_(high) is a ratio of high frequency pixels to the wholeimage at the current resolution level r, and n is a predeterminedparameter.

Subsequently, the partial differential coefficient calculation unit 107Bcalculates three partial differential coefficients f_(x)(x, y), f_(y)(x,y) and f_(t)(x, y) required in motion vector estimation processing fromthe images f and f′ at the current resolution level r.

Then, the motion vector estimation unit 109B executes motion vectorestimation processing in order of the stages 1 to 3.

Specifically, first in the stage 1, the motion vector estimation unit109B conducts repetitive calculations for high frequency pixels (whitepixels in the high frequency mask image in FIG. 5) and optimizes anestimated motion vector for high frequency pixels. This stage isexecuted from a repetition of the first time to a repetition of thei^(r) ₁-th time. Subsequently, in the stage 2, the motion vectorestimation unit 109B conducts repetitive calculations for low frequencypixels (black pixels in the high frequency mask image in FIG. 5) andoptimizes an estimated motion vector for low frequency pixels. Thisstage is executed from a repetition of the (i^(r) ₁+1)-st time to arepetition of the i^(r) ₂-th time. Finally, in the stage 3, the motionvector estimation unit 109B conducts repetitive calculations for allpixels in the image and optimizes an estimated motion vector for allpixels.

By the way, the motion vector estimation processing in each stage isconducted by using the technique disclosed in NPL 1.

A motion vector for all pixels at the resolution level r is estimated bythe processing described heretofore.

Finally, the motion upscaling unit 113 upscales estimated motion vectorimages u and v to a resolution of twice by bilinear interpolation, andthereby generates an initial value of an estimated motion vector at thenext resolution level r+1.

A motion vector for all pixels at the current resolution between thecurrent frame and the next frame is estimated by repeating theabove-described processing up to the resolution level 3.

Example 2

The example 1 is directed for a case where the input is a progressiveimage. If it is supposed to use the motion vector estimation in theexample 1, it is necessary in a case where the input is an interlaceimage to decompose images into even-numbered field images andodd-numbered field images, convert each field image to a progressiveimage by conducting, for example, processing of interpolating a pixelvalue on a line that does not exist in each field image (an odd-numberedline in the case of the even-numbered field image and an even-numberedline in the case of the odd-numbered field image) with an average ofpixel values on lines existing above and below the line that does notexist, and apply the motion vector estimation in the example 1 to twoprogressive images.

In this technique, however, a motion is estimated not only for pixelsexisting in each field image but also for pixels generated by theinterpolation. A calculation quantity that is twice a calculationquantity originally needed occurs.

In view of this point, an example 2 is provided. The example 2 relatesto a motion compensation type interlace-progressive (IP) conversion inwhich in a case where the input is an interlace image, motion vectorestimation is conducted without converting an interlace image to aprogressive image as pre-processing and then a progressive image isgenerated on the basis of the estimated motion vector.

By the way, the example 2 is a concrete example of the embodiment 3.

A state of the example 2 is illustrated in FIG. 10. In the example 2, aprogressive image is generated by receiving an interlace image as aninput, estimating a motion vector between an even-numbered field and anodd-numbered field in the interlace image, conducting motioncompensation on an odd-numbered field image on the basis of theestimated motion vector, and uniting the odd-numbered field subjected tothe motion compensation with the even-numbered field image.

The example 2 differs from the example 1 in that processing ofdecomposing an input interlace image into an even numbered field imagef_(even) and an odd-numbered field image f_(odd), processing ofconducting motion compensation on the odd-numbered field image f_(odd)by using a motion vector estimation result, and processing of unitingthe even numbered field image f_(even) and the odd-numbered field imagef_(odd) subjected to motion compensation into one progressive image areadded and Math. used in motion vector estimation processing differs.

Hereafter, Math. used in the motion vector estimation processing will bedescribed.

Math. used in the motion vector estimation processing in the example 2is as follows.

Math. 8

2 f_(x)^(even)(x, y)² ⋅ u(x, y) + 2 f_(x)^(even)(x, y) ⋅ f_(y)^(even)(x, y) ⋅ v(x, y) + 2 f_(x)^(even)(x, y) ⋅ f_(t)^(even)(x, y) − f_(x)^(even)(x, y) ⋅ f_(y)^(even)(x, y) − α ⋅ Δ u(x, y) = 0

2 f_(x)^(even)(x, y) ⋅ f_(y)^(even)(x, y) ⋅ u(x, y) + 2 f_(y)^(even)(x, y)² ⋅ v(x, y) + 2 f_(y)^(even)(x, y) ⋅ f_(t)^(even)(x, y) − f_(y)^(even)(x, y)² − α ⋅ Δ v(x, y) = 0

By the way, in a case where the input image includes a telop or the likethat scrolls at a constant velocity in the lateral direction orlongitudinal direction, either one or both of u(x, y) and v(x, y)becomes known. For example, in a case where a telop that scrolls in thelateral direction in a bottom part of a screen exists, it is alreadyknown that v(x, y)=0 in pixels in the bottom part of the screen.

In the case where it is previously known that u(x, y) and v(x, y) assumespecific values u₀(x, y) and v₀(x, y) in this way,w _(u)(x,y)·(u(x,y)−u ₀(x,y))w _(v)(x,y)·(v(x,y)−v ₀(x,y))are added to right sides of the first and second expressions,respectively. Here, w_(u)(x, y) and w_(v)(x, y) represent reliabilitieswhether the coordinate (x, y) assumes u₀(x, y) and v₀(x, y),respectively. As the reliability becomes higher, w_(u)(x, y) andw_(v)(x, y) assume larger values. In a case where there is noreliability, i.e., it is not previously known to assume a specificvalue, w_(u)(x, y) and w_(v)(x, y) become zero.

By the way, in the example 2, motion compensation is conducted on theodd-numbered field image by taking the even-numbered field as areference. In a case where motion compensation is conducted on theeven-numbered field image by taking the odd-numbered field as areference, however, it suffices to interchange f_(even) and f_(odd) witheach other in the expressions.

In a case where the input is an interlace image, it becomes possible toestimate a motion between field images with a minimum requiredcalculation quantity by using the example 2.

Example 3

In the example 1 and the example 2, a motion vector every pixel isfound. However, a motion vector every pixel group including at least twopixels may be found. In the example 1 and the example 2, the number ofpixels included in a pixel group is 1.

By the way, the above-described motion vector estimation device can beimplemented by hardware, software, or a combination of them.Furthermore, a motion vector estimation method conducted by theabove-described motion vector estimation device or another device canalso be implemented by hardware, software, or a combination of them.Here, “implemented by software” means “implemented by a computer thatreads a program and executes the program.”

The program can be stored in non-transitory computer readable media ofvarious types and supplied to a computer. The non-transitory computerreadable media include tangible storage media of various types. Examplesof the non-transitory computer readable media include magnetic recordingmedia (for example, a flexible disc, magnetic tape, and a hard discdrive), magneto-optical recording media (for example, a magneto-opticaldisc), a CD-ROM (Read Only Memory), a CD-R, a CD-R/W, a semiconductormemory (for example, a mask ROM, and a PROM (Programmable ROM), an EPROM(Erasable PROM), a flash ROM, and a RAM (random access memory).Furthermore, the program may be supplied to a computer by transitorycomputer readable media of various types. Examples of the transitorycomputer readable media include an electric signal, an optical signal,and electromagnetic waves. The transitory computer readable media cansupply the program to a computer via a wired communication path such asan electric wire or an optical fiber, or a wireless communication path.

A part or the whole of the above-described embodiments can be stated asin the ensuing additions, but is not restricted to the ensuingadditions.

(Addition 1)

A motion vector estimation device that estimates a motion vector foreach of a plurality of pixel groups included in an input image, eachpixel group including at least one pixel, by repetitive calculations,

the motion vector estimation device comprising:

a means that conducts the repetitive calculations intended for pixelgroups having a high frequency component among the plurality of pixelgroups included in the input image, and then conducts the repetitivecalculations intended for pixel groups having no high frequencycomponents among the plurality of pixel groups included in the inputimage.(Addition 2)

The motion vector estimation device according to addition 1, furthercomprising a means that conducts the repetitive calculations intendedfor pixel groups having no high frequency components among the pluralityof pixel groups, and then conducts the repetitive calculations intendedfor a whole of the plurality of pixel groups.

(Addition 3)

The motion vector estimation device according to addition 1 or 2,further comprising:

a means that generates a primary low resolution image which is lower inresolution than the input image, on the basis of the input image; and

a means that estimates a motion vector for the primary low resolutionimage, and then estimates a motion vector for the input image by usingthe motion vector estimated for the primary low resolution image as aninitial value.

(Addition 4)

The motion vector estimation device according to addition 3, furthercomprising:

a means that generates a secondary low resolution image which is lowerin resolution than the primary low resolution image, on the basis of theprimary low resolution image; and

a means that estimates a motion vector for the secondary low resolutionimage, and then estimates a motion vector for the primary low resolutionimage by using the motion vector estimated for the secondary lowresolution image as an initial value.

(Addition 5)

The motion vector estimation device according to any one of additions 1to 4, further comprising a means that determines whether each of aplurality of pixel groups has a high frequency component.

(Addition 6)

The motion vector estimation device according to any one of additions 1to 5, wherein the repetitive calculations are repetitive calculationsfor minimizing an energy function that is used to estimate the motionvector and that includes a data term and a regularizing term.

(Addition 7)

An interlace-progressive conversion device comprising:

the motion vector estimation device according to any one of additions 1to 6, an odd-numbered field image or an even-numbered field image in aninterlace image being supplied to the motion vector estimation device asthe input image;

a motion compensation means that conducts motion compensation on theodd-numbered field image or the even-numbered field image by using amotion vector estimated by the motion vector estimation device; and

a field combination means that combines the odd-numbered field image orthe even-numbered field image subjected to the motion compensation withthe other field image.

(Addition 8)

A motion vector estimation method used to estimate a motion vector foreach of a plurality of pixel groups included in an input image, eachpixel group including at least one pixel, by repetitive calculations,

the motion vector estimation method comprising:

a step of conducting the repetitive calculations intended for pixelgroups having a high frequency component among the plurality of pixelgroups included in the input image, and then conducting the repetitivecalculations intended for pixel groups having no high frequencycomponents among the plurality of pixel groups included in the inputimage.(Addition 9)

The motion vector estimation method according to addition 8, furthercomprising a step of conducting the repetitive calculations intended forpixel groups having no high frequency components among the plurality ofpixel groups, and then conducting the repetitive calculations intendedfor a whole of the plurality of pixel groups.

(Addition 10)

The motion vector estimation method according to addition 8 or 9,further comprising:

a step of generating a primary low resolution image which is lower inresolution than the input image, on the basis of the input image; and

a step of estimating a motion vector for the primary low resolutionimage, and then estimating a motion vector for the input image by usingthe motion vector estimated for the primary low resolution image as aninitial value.

(Addition 11)

The motion vector estimation method according to addition 10, furthercomprising:

a step of generating a secondary low resolution image which is lower inresolution than the primary low resolution image, on the basis of theprimary low resolution image; and

a step of estimating a motion vector for the secondary low resolutionimage, and then estimating a motion vector for the primary lowresolution image by using the motion vector estimated for the secondarylow resolution image as an initial value.

(Addition 12)

The motion vector estimation method according to any one of additions 8to 11, further comprising a step of determining whether each of aplurality of pixel groups has a high frequency component.

(Addition 13)

The motion vector estimation method according to any one of additions 8to 12, wherein the repetitive calculations are repetitive calculationsfor minimizing an energy function that is used to estimate the motionvector and that includes a data term and a regularizing term.

(Addition 14)

An interlace-progressive conversion method comprising:

the steps in the motion vector estimation method according to any one ofadditions 8 to 13, an odd-numbered field image or an even-numbered fieldimage in an interlace image being supplied to the motion vectorestimation method as the input image;

a motion compensation step of conducting motion compensation on theodd-numbered field image or the even-numbered field image by using amotion vector estimated in the motion vector estimation method; and

a field combination step of combining the odd-numbered field image orthe even-numbered field image subjected to the motion compensation withthe other field image.

(Addition 15)

A motion vector estimation program for causing a computer to function asa motion vector estimation device that estimates a motion vector foreach of a plurality of pixel groups included in an input image, eachpixel group including at least one pixel, by repetitive calculations,

the motion vector estimation program causing the computer to function asa means that conducts the repetitive calculations intended for pixelgroups having a high frequency component among the plurality of pixelgroups included in the input image, and then conducts the repetitivecalculations intended for pixel groups having no high frequencycomponents among the plurality of pixel groups included in the inputimage.(Addition 16)

The motion vector estimation program according to addition 15, forfurther causing a computer to function as a means that conducts therepetitive calculations intended for pixel groups having no highfrequency components among the plurality of pixel groups, and thenconducts the repetitive calculations intended for a whole of theplurality of pixel groups.

(Addition 17)

The motion vector estimation program according to addition 15 or 16, forfurther causing a computer to function as:

a means that generates a primary low resolution image which is lower inresolution than the input image, on the basis of the input image; and

a means that estimates a motion vector for the primary low resolutionimage, and then estimates a motion vector for the input image by usingthe motion vector estimated for the primary low resolution image as aninitial value.

(Addition 18)

The motion vector estimation program according to addition 17, forfurther causing a computer to function as:

a means that generates a secondary low resolution image which is lowerin resolution than the primary low resolution image, on the basis of theprimary low resolution image; and

a means that estimates a motion vector for the secondary low resolutionimage, and then estimates a motion vector for the primary low resolutionimage by using the motion vector estimated for the secondary lowresolution image as an initial value.

(Addition 19)

The motion vector estimation program according to any one of additions15 to 18, for further causing a computer to function as a means thatdetermines whether each of a plurality of pixel groups has a highfrequency component.

(Addition 20)

The motion vector estimation program according to any one of additions15 to 19, wherein the repetitive calculations are repetitivecalculations for minimizing an energy function that is used to estimatethe motion vector and that includes a data term and a regularizing term.

(Addition 21)

An interlace-progressive conversion program for causing a computer tofunction as an interlace-progressive conversion device, theinterlace-progressive conversion program causing the computer tofunction as:

means in the motion vector estimation device according to any one ofadditions 1 to 6, an odd-numbered field image or an even-numbered fieldimage in an interlace image being supplied to the motion vectorestimation device as the input image;

a motion compensation means that conducts motion compensation on theodd-numbered field image or the even-numbered field image by using amotion vector estimated by the motion vector estimation device; and

a field combination means that combines the odd-numbered field image orthe even-numbered field image subjected to the motion compensation withthe other field image.

The present application is based upon Japanese Patent Application No.2011-249679 (filed on Nov. 15, 2011), or claims priority based uponJapanese Patent Application No. 2011-249679 according to the Treaty ofParis. Contents disclosed in Japanese Patent Application No. 2011-249679are incorporated in the present specification by referring to JapanesePatent Application No. 2011-249679.

Representative embodiments of the present invention have been describedin detail. It is to be understood that various changes, substitutions,and alternatives can be made without departing from the spirit andscopes of the invention defined in claims. Furthermore, the inventorsintend that equivalent scope of the claimed invention is maintained evenif claims are corrected in application procedures.

INDUSTRIAL APPLICABILITY

The motion vector estimation device, motion vector estimation method,and motion vector estimation program according to the present inventionare industrially useful, because a motion vector can be estimated withless calculation quantity.

REFERENCE SIGNS LIST

-   101, 101B Motion vector estimation device-   103, 103B High frequency/low frequency decision unit-   105, 105B Number of times of repetition determination unit-   107, 107B Partial differential coefficient calculation unit-   109, 109B Motion vector estimation unit-   111 Resolution pyramid creation unit-   113 Motion vector upscaling unit

What is claimed is:
 1. A motion vector estimation device that estimatesa motion vector for each of a plurality of pixel groups included in aninput image, each pixel group including at least one pixel, byrepetitive calculations, the motion vector estimation device comprising:a unit configured to conduct the repetitive calculations intended forpixel groups having a high frequency component among the plurality ofpixel groups included in the input image, and then conduct therepetitive calculations intended for pixel groups having no highfrequency components among the plurality of pixel groups included in theinput image.
 2. The motion vector estimation device according to claim1, further comprising a unit configured to conduct the repetitivecalculations intended for pixel groups having no high frequencycomponents among the plurality of pixel groups, and then conduct therepetitive calculations intended for a whole of the plurality of pixelgroups.
 3. The motion vector estimation device according to claim 1,further comprising: a unit configured to generate a primary lowresolution image which is lower in resolution than the input image, onthe basis of the input image; and a unit configured to estimate a motionvector for the primary low resolution image, and then estimates a motionvector for the input image by using the motion vector estimated for theprimary low resolution image as an initial value.
 4. The motion vectorestimation device according to claim 3, further comprising: a unitconfigured to generate a secondary low resolution image which is lowerin resolution than the primary low resolution image, on the basis of theprimary low resolution image; and a unit configured to estimate a motionvector for the secondary low resolution image, and then estimates amotion vector for the primary low resolution image by using the motionvector estimated for the secondary low resolution image as an initialvalue.
 5. The motion vector estimation device according to claim 1,further comprising a unit configured to determine whether each of aplurality of pixel groups has a high frequency component.
 6. The motionvector estimation device according to claim 1, wherein the repetitivecalculations are repetitive calculations for minimizing an energyfunction that is used to estimate the motion vector and that includes adata term and a regularizing term.
 7. An interlace-progressiveconversion device comprising: the motion vector estimation deviceaccording to claim 1, an odd-numbered field image or an even-numberedfield image in an interlace image being supplied to the motion vectorestimation device as the input image; a motion compensation unitconfigured to conduct motion compensation on the odd-numbered fieldimage or the even-numbered field image by using a motion vectorestimated by the motion vector estimation device; and a fieldcombination unit configured to combine the odd-numbered field image orthe even-numbered field image subjected to the motion compensation withthe other field image.
 8. A motion vector estimation method used toestimate a motion vector for each of a plurality of pixel groupsincluded in an input image, each pixel group including at least onepixel, by repetitive calculations, the motion vector estimation methodcomprising: a step of conducting the repetitive calculations intendedfor pixel groups having a high frequency component among the pluralityof pixel groups included in the input image, and then conducting therepetitive calculations intended for pixel groups having no highfrequency components among the plurality of pixel groups included in theinput image.
 9. The motion vector estimation method according to claim8, further comprising a step of conducting the repetitive calculationsintended for pixel groups having no high frequency components among theplurality of pixel groups, and then conducting the repetitivecalculations intended for a whole of the plurality of pixel groups. 10.The motion vector estimation method according to claim 8, furthercomprising: a step of generating a primary low resolution image which islower in resolution than the input image, on the basis of the inputimage; and a step of estimating a motion vector for the primary lowresolution image, and then estimating a motion vector for the inputimage by using the motion vector estimated for the primary lowresolution image as an initial value.
 11. The motion vector estimationmethod according to claim 10, further comprising: a step of generating asecondary low resolution image which is lower in resolution than theprimary low resolution image, on the basis of the primary low resolutionimage; and a step of estimating a motion vector for the secondary lowresolution image, and then estimating a motion vector for the primarylow resolution image by using the motion vector estimated for thesecondary low resolution image as an initial value.
 12. The motionvector estimation method according to claim 8, further comprising a stepof determining whether each of a plurality of pixel groups has a highfrequency component.
 13. The motion vector estimation method accordingto claim 8, wherein the repetitive calculations are repetitivecalculations for minimizing an energy function that is used to estimatethe motion vector and that includes a data term and a regularizing term.14. An interlace-progressive conversion method comprising: the steps inthe motion vector estimation method according to claim 8, anodd-numbered field image or an even-numbered field image in an interlaceimage being supplied to the motion vector estimation method as the inputimage; a motion compensation step of conducting motion compensation onthe odd-numbered field image or the even-numbered field image by using amotion vector estimated in the motion vector estimation method; and afield combination step of combining the odd-numbered field image or theeven-numbered field image subjected to the motion compensation with theother field image.
 15. A non-transitory computer readable medium storinga motion vector estimation program for causing a computer to function asa motion vector estimation device that estimates a motion vector foreach of a plurality of pixel groups included in an input image, eachpixel group including at least one pixel, by repetitive calculations,the motion vector estimation program causing the computer to function asa unit configured to conduct the repetitive calculations intended forpixel groups having a high frequency component among the plurality ofpixel groups included in the input image, and then conduct therepetitive calculations intended for pixel groups having no highfrequency components among the plurality of pixel groups included in theinput image.
 16. The non-transitory computer readable medium storing themotion vector estimation program according to claim 15, for furthercausing a computer to function as a unit configured to conduct therepetitive calculations intended for pixel groups having no highfrequency components among the plurality of pixel groups, and thenconduct the repetitive calculations intended for a whole of theplurality of pixel groups.
 17. The non-transitory computer readablemedium storing the motion vector estimation program according to claim15, for further causing a computer to function as: a unit configured togenerate a primary low resolution image which is lower in resolutionthan the input image, on the basis of the input image; and a unitconfigured to estimate a motion vector for the primary low resolutionimage, and then estimates a motion vector for the input image by usingthe motion vector estimated for the primary low resolution image as aninitial value.
 18. The non-transitory computer readable medium storingthe motion vector estimation program according to claim 17, for furthercausing a computer to function as: a unit configured to generate asecondary low resolution image which is lower in resolution than theprimary low resolution image, on the basis of the primary low resolutionimage; and a unit configured to estimate a motion vector for thesecondary low resolution image, and then estimates a motion vector forthe primary low resolution image by using the motion vector estimatedfor the secondary low resolution image as an initial value.
 19. Thenon-transitory computer readable medium storing the motion vectorestimation program according to claim 15, for further causing a computerto function as a unit configured to determine whether each of aplurality of pixel groups has a high frequency component.
 20. Thenon-transitory computer readable medium storing the motion vectorestimation program according to claim 15, wherein the repetitivecalculations are repetitive calculations for minimizing an energyfunction that is used to estimate the motion vector and that includes adata term and a regularizing term.